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(57) ABSTRACT 

The present invention provides apparatus for, and novel 
methods of representing the content of a musical score in an 
animated format that graphically prompts the user of a 
keyboard instrument. The graphic format elucidates features 
of the score in a way that may also be useful as a supplement 
to conventional keyboard instruction. The animation is dis- 
played on the screen of a computer monitor together with a 
live video image of the user's hands on a musical keyboard. 
Vicv^ring only the images on the monitor, the user guides his 
fingers to the keys targeted by the animation. He is thiis able 
lo play the proper keys on the real keyboard without 
diverting his eyes from the monitor. In the preferred 
embodiment, the system provides a real time display of both 
the score information, keyboard and the user's hand place- 
ment relative to the keyboard. Preferably, the display of the 
musical score includes a note position depicted in a hori- 
zontal relationship to the keyboard, and includes a temporal 
indication for the notes. 

31 Claims, 6 Drawing Sheets 
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COMPUTER GRAPHIC ANIMATION, LIVE temperament whole tone graphic/tactile keyboard and a 

VIDEO INTERACTIVE METHOD FOR nomographic system of music notation. The nomographic 

PLAYING KEYBOARD MUSIC notation is correlated to the keyboard by nomographically 

marking the lines and spaces of the stave adjacent the key 

RELATED APPU CATION INFORMATION s signature and similarly graphically marking the notes played 

^ , . , . . . r • • 1 . * on the upper or lower row with a diagonal slant. 

This application claims priority from provisional patent . . . . , ^ , r * 

application 60/169,325, filed Dec. 6, 1999, entitled "Com- ^.S. Pat. No. 5,153,829 issued to Furuya, et al sets forth 

puL Graphic Ani;nali;n-Live Video Int;ractive Method ^i^^'T'^^?^^^^^^^ INFORMATION PRO- 

for Playing Keyboard Music''. CESSING APPARATUS which includes: a umt for display- 

10 ing on a screen a musical score, keyboard and tone time 
FILED OF THE INVENTION information to be inputted; a unit for designating the posi- 
tion of the keyboard, and tone time information, respectively 
This invention relates generally to systems and methods displayed on the display unit; a unit for storing musical 
for facilitating playing music. More particularly, the inven- information produced through designation by the designat- 
tion relates to systems, apparatus and methods for providing 15 ^^^^ j^e position of the keyboard and tone time 
a graphical animation for use in playing musical information displayed on the display unit; a unit for con- 
instruments, particularly keyboard instruments. trolling the display of the musical score, keyboard and tone 
T> AOTr*-n/MTKTT^ r^T: T^TT T KFi TT? Km r^xT time information on the screen of the display unit, and for 
BACKGROUND OF THE INVENTION controUing the display of a pattern of musical tone or rest on 
The invention relates generally to automated multimedia ^-O (he musical score on the display unit in accordance with the 
systems, and, more specifically, to a computer-based system position of the keyboard and tone time information respec- 
for prompting the player of a musical keyboard. lively designated by the designating unit; and a unit for 
Playing a musical keyboard can be an enjoyable cxperi- generating a musical tone by reading the musical informa- 
ence. Most people have attempted it, but a relative few have ^on stored m the storage unit. 

mastered it, largely because of the initial difficulty and the U.S. Pat. No. 5,315,911 issued to Ochi sets forth MUSIC 

hours of dedicated study and practice required. The many SCORE DISPLAY DEVICE including a storage device for 

competing claims on a person's time usually leave insuffi- storing music score data, a display device for displaying the 

cient room for the regimen of keyboard practice. Parents music score data for each specified section, and a compara- 

who spend thousands of dollars on piano lessens for their tor for comparing the musical playing data supplied from a 

children, often see their youngsters quit in frustration after player with the music score data. The music score display 

a couple of years of struggle. device further has a retrieval device for retrieving a playing 

Various schemes, often based on computers, have been position according to the compared result of the comparator, 

devised to assist with the learning process. Some of this and a display controller for controlling display section of the 

software is very good at providing instruction in traditional display device based on the retrieval result of the rctncval 

keyboard methods in which the student learns to read the device. 

notes on the staff of a musical score, whether it be printed U.S. Pat. Nos. 5,533,903 and 5,746,605 issued to 

or displayed on a computer monitor. For example, Yamaha Kennedy set forth METHOD AND SYSTEM FOR MUSIC 

Corporation produces "Disklavier Pro 2000" including an TRAINING including a method of and computer system for 

LCD screen to view digital sheet music and video music training a student to play a musical work having an identi- 

perforraances that sync to the moving keys. fiable rhythm structure, the musical work being decompos- 

A major obstacle remaining for many students of the able into a plurality of variations each maintaining the 

traditional method, however, is the large amount of continu- identifiable rhythm structure. According to the method, a 

ous information processing required of the brain to translate musical score of each variation is generated and tiien the 

the arcane notation of printed music into the corresponding 45 musical scores are arranged in order of rhythmic difficulty, 

motor commands to the fingers. Keeping track of sharps and An animated movie is then generated, the movie comprising 

tiats amid a stream of chromatic signs, recognizing notes the music score of at least one variation, a sound recording 

well above or below the staff, and executing of all the of the music score, a graphic device that highlights the 

various notational nuances can be particularly daunting. identifiable rhythm structure and a representation of how the 

Some schemes attempt to address this problem by using 50 '"^^^^ P^^^^^ ' "'"f^'^^^ instrument. The 
a system of lights near or inside of the keys to indicate which animated movie is then presented to the student, 
keys are to be played al a given time, but this approach can U.S. Pat, No. 5,583,308 issued to Owen sets forth MUSI- 
only crudely guide the performance. The Ughts are unable to CAL EFFECTS APPARATUS AND TONE CONTROL 
convey much of the auxiliary information coded in a piano PROCESS FOR A MUSICAL INSTRUMENT. In an auto- 
score. There arc about a dozen primary parameters that can 55 mated system for learning to play a musical instrunient, 
be associated with a note to carry instructions as to how that music data and tone control data for a musical composition 
note should be played. A key's light being either on or off of an exemplary musical instrument is stored in machine 
can transmit only a fraction of this information. Another readable form. An audio signal generated by a musical 
serious shortcoming of these methods is the inability of the instrument being played, corresponding to the exemplary 
user to scan upstream in the flow of approaching notes. This musical instrument, is inputted through a musical effects 
deficiency impairs his ability to smoothly position his hands apparatus wherein the musical tone of the musical instru- 
in anticipation of notes drawing near their instant of play. ment being played is mixed with the stored music data. The 
Prior art with relevance to the present invention includes: musical effects apparatus is utilized, under digital control, to 

U.S. Pat. No. 4,926,734 issued to Rickey sets forth match the musical tone of the musical instrument being 

GRAPHIC/TACTILE MUSICAL KEYBOARD AND 65 played with that of the exemplary musical mslmment. 

NOMOGRAPHIC MUSIC NOTATION in which a graphic/ U.S. Pat. No. 5,585,583 issued to Owen sets forth INTER- 

tactile musical instruction system includes a three-row equal ACTIVE MUSICAL INSTRUMENT INSTRUCTION 
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SYSTEM in which a musical iostrument instruction system 
utilizes a CD-ROM electronic storage meditim to store audio 
and video musical instrument instructional materials. 
Instructional materials are accessed utilizing a computer 
which provides an interactive system in which animation, 5 
voice, music, video and imagery are integrated in essentially 
real time and under complete control of the music student. 
The system permits the student to access the instructional 
materials in a variety of ways, and control the parameters 
governing the presentation of the instructional materials, 
including control of the tempo of the music and whether or 
not displays of the maimer of playing the musical instrument 
will be shown with a displayed notation or tablature relating 
to the selected musical piece. A mixer is provided for 
combining musical output of the practice instrument with 
musical output generated by the computer corresponding to 
the selected musical piece. 

U.S. Pat. No. 5,690,496 issued to Kennedy sets forth 
MULTIMEDIA PRODUCT FOR USE IN A COMPUTER 
FOR MUSIC INSTRUCTION AND USE in which a com- ^ 
puter is used to assist a student to learn to play a given 
musical work previously recorded on a master recording. A 
method of making the product is described which ensures 
that multimedia representations of the work are properly 
synchronized to the actual tempo variations, if any, in the ^5 
master recording. A method of using the product for music 
training is also described. 

U.S. Pat. No. 5,728,960 issued to Sitrick sets forth 
MULTI-DIMENSIONAL TRANSFORMATION SYS- 
TEMS AND DISPLAY COMMUNICATION ARCHITEC- 30 
TURE FOR MUSICAL COMPOSITIONS in which a musi- 
cal display system produces a display presentation in one of 
a plurality of formats from an original musical composition, 
the formats being created from the original musical compo- 
sition responsive to input variables and a selected operating 35 
mode. The system is comprised of means to receive the 
original composition, a memory, a selection subsystem, a 
controller, and a display subsystem. The display system 
provides for selection of original compositions, creation of 
derivative compositions, distribution of compositions, moni- 40 
toring of each performer's performance, and also allows for 
retrieval and editing of musical compositions, such as 
changing keys, pitch, tempo, and many other parameters, 

U.S. Pat. No. 5,908,997 issued to Arnold, ct al. sets forth 
ELECTRONIC MUSIC INSTRUMENT SYSTEM WITH 45 
MUSICAL KEYBOARD comprising: an electronic music 
instrument, having selectable groups of reproducible sounds 
and individually selectable reproducible sounds; a signal 
generator for energizing an audio amplifier responsive to 
different digital audio sources, including the groups of 50 
sounds and the individual sounds a graphical user interface 
for displaying at least one control graphic representing 
controllable parameters of the audio signals generated by the 
generator; and, a controller responsive to operation of the 
control graphic for adjusting the controllable parameters of ss 
the generator and for selectively coupling different ones of 
the sources to the generator. The graphical user interface can 
comprise: a video display; and, a touch-responsive overlay. 
The controllable parameters can include multiple instrument 
sound selection and sound layer assignment, controlled 60 
responsive to operation of the at least one control graphic. 

U.S. Pat. No. 6,042,555 issued to Kramer, el al. sets forth 
FORCE-FEEDBACK INTERFACE DEVICE FOR THE 
HAND in which a man-machine interface is disclosed which 
provides force information to sensing body parts. The inter- 65 
face is comprised of a force-generating device (106) that 
produces a force which is transmitted to a force -applying 
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device via force -transmitting means. For example a glove 
employing force feedback may be programmed to teach 
finger dexterity, finger timing and even the motions neces- 
sary to learn some musical instruments. If the user were 
learning the piano, as fingers are flexed, the user would 
receive fingertip pressure from virtual keys signifying to the 
user that he had pressed the key, 

U.S. Pat. No. 6,066,791 issued to Renard, et al. seU forth 
SYSTEM FOR INSTRUCTING THE PLAYING OF A 
MUSICAL INSTRUMENT which discloses a novel method 
and apparatus for instructing a student how to play a musical 
instrument. The method includes providing a display device, 
displaying a staff on the display device, displaying an image 
on the display device, and instructing the student to focus on 
the image whUe preferably using a musical instnmient to 
play the notes on the staff, wherein the image moves 
generally in the direction of the progression of notes on the 
staff while the student attempts to play the, notes with the 
musical instrument. As the image moves, the image prefer- 
ably moves in a direction generally parallel to the staff. 

Although the prior art addresses musical keyboard play- 
ing in various ways, none includes a live video image of the 
player's hands interacting with on-screen animation. For 
those who would like to read keyboard music and play it 
with their own hands, but who would welcome a simpler 
alternative to mastering the skill of sight reading a musical 
score, there is a need for a system that can reduce the amount 
of mental information processing involved, while preserving 
aU of the information a score contains. The present invention 
fulfills this need, and provides other distinct advantages. 

SUMMARY OF THE INVENTION 

The present invention provides a method of representing 
the content of a musical score in an animated format that 
graphically prompts the user of a keyboard instrument. 
Intended primarily for pleasure, the graphic format eluci- 
dates features of the score in a way that may also be useful 
as a supplement to conventional keyboard instruction. The 
animation is displayed on the screen of a computer monitor 
together with a live video image of the user's hands on a 
musical keyboard. Viewing only the images on the monitor, 
the user guides his fingers to the keys targeted by the 
animation. He is thus able to play the proper keys on the real 
keyboard without diverting his eyes from the monitor. At the 
same time, he bypasses the intense mental information 
processing involved in reading printed sheet music. 

In the preferred embodiment the monitor is supported at 
eye level near the position otherwise occupied by sheet 
music. A video camera is mounted about four feet above the 
center of the keyboard, and is directed downward. The 
resulting keyboard image is displayed at the bottom and 
across the full width of the monitor screen. Across the top of 
the screen is a tool bar for controlling and monitoring the 
displayed material. The mid -screen area is devoted to the 
animation. Here the horizontal co-ordinate represents posi- 
tion along the keyboard in one-to-one correspondence with 
the bordering keyboard image below. The vertical 
co-ordinate represents time, increasing from bottom to top. 
Objects higher on the screen represent events that occur later 
than objects lower on the screen. The present time is defined 
as the lower edge of the animation area where it borders the 
keyboard image. All objects in the animation area move 
steadily downward until they touch the border, at which 
point they have arrived at the present. 

A musical note is represented by a "sprite" typically 
elliptical in shape, initially appearing directly above, and 
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moving down-screen toward the image of the corresponding 
key. The distance of the sprite from the key image is a 
measure of the time remaining until the key is to be played. 
This shrinking distance gives the user time to take aim at the 
targeted key, and to anticipate the moment of play. 5 

To keep the screen uncluttered, a sprite does not appear 
until the corresponding note is within one beat of play. The 
vertical length of the sprite represents the duration of the 
note. A quarter-note sprite, for example, is twice as long as 
an eighth-note sprite. The key is played when the sprite first jq 
touches the key image. While maintaining the same down- 
ward speed, the sprite then appears to slide behind the key 
image, and the key is held until the last of the sprite 
disappears from view. 

Through the use of color, shape, and other features, the ^5 
sprites are designed to carry all of the information contained 
in the musical score. For example blue sprites represent 
notes to be played with the left hand, and green with the right 
hand. Both colors switch to red at the instant of contact with 
the key image to announce the moment of play. The degree 20 
of blue or green color saturation indicates Uie loudness of a 
note, ranging from pale colors for pianissimo, to rich colors 
for fortissimo. Pedaling is represented by a black bottom 
edge on a sprite for pedal down, and a black upper edge for 
pedal up. 25 

The style of attack is indicated by the degree to which the 
leading and trailing edges of the elliptical sprite are clipped, 
ranging from no clipping for the smoothly joined notes of 
legato, through progressively more severe clipping for por- 
tamento and staccato. A totally squared off leading edge 30 
indicates the "hammered" note requested by a marteUato. 

Fingering is suggested for each note by an Arabic numeral 
within a small circle of the same color as the approaching 
sprite, and fixed in position with its lowest point in contact 
with the targeted key. The colored circle helps to identify the 35 
key while the sprite is still some distance away. 
The circle appears simultaneously with the sprite, and dis- 
appears when touched by the sprite. 

The various parameters for a note are included as fields of 
a record in a "piece file", which is an array of such records 40 
for all of the notes in a particular piece. The first record in 
the array is a dummy note with its fields containing initial- 
ization information such as the time signature, default met- 
ronome setting, and total number of measures in the piece. 
A library of piece files is made available for downloading 45 
from the Internet, on CD-ROM or other storage medium for 
loading into computer memory. 

The user exercises control over the animation from two 
access points. The first is the Set-Up Screen that appears just 
after an Opening Screen when the program is launched. The 50 
Set-Up Screen is where the user chooses a piece from the 
library, and initializes the fundamental operating 
parameters, such as the metronome setting, start and stop 
points, the magnitude of tempo variations, and whether to 
repeal the selected measures when finished. There he also 55 
selects from a series of options for audio features. For some 
pieces there will be a choice of the style of different artists, 
as reflected in the tempo variations and treatment of loud and 
soft passages. The user can also return to the Set-Up screen 
at any time to change these settings. The second control 60 
point is the monitoring and control tool bar at the top of the 
Action Screen where the animation is displayed. These 
controls let the user jump backward or forward in the piece, 
start, stop, or pause the action, etc. 

The invention does not magically eliminate the need for 65 
practice. One still must play a piece many times to become 
proficient, but the mental effort is greatly reduced. Freed of 
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the burden of reading the sheet music, the user can concen- 
trate on training his fingers to quickly assume the indicated 
patterns. The user's interaction with the animation is rather 
like that of a video game, and practice is fun. Because it is 
fun, it happens, and rapid progress is the result. 

The accompanying drawings and detailed description 
offer further clarification of the features and advantages of 
the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 shows a block diagram of the components of one 
implementation of this invention. 

FIG. 2 shows a flow chart of an implementation of this 
invention. 

FIG. 3 shows a planned view of a graphical display for 
interface with the user. 

FIG. 4 shows a planned view of the display in operation. 

FIG. 5 shows a planned view of the display of FIG. 4, but 
at a later point in time. 

FIG. 6 shows a graphical display of the image of FIG. 4 
and FIG, 5, but at a yet later point in time. 

DETAILED DESCRIPTION 

FIG. 1 shows a block diagrammatic view of the compo- 
nents of the system in the preferred embodiment. A com- 
puter 102 is provided to effectuate control of the system. The 
prototype is a conventional IBM-compatible PC with a 133 
MHz Pentium processor, 16 MB of RAM and a 1 .9 GB hard 
drive. Included is a "frame grabber*' card to take input from 
a video camera, and software for sizing and positioning the 
associated video image on the monitor. (A more portable 
alternative would be a laptop computer with a universal 
serial bus (USB) port, which can also be configured to 
accept input from a video camera.) In yet another simpler 
embodiment, an LED array may be used, and optionally, 
integrated with the keyboard instrument. As is required for 
the operation of the computer 102, the system may include 
a mass storage type device, e.g., a CD ROM drive 104, and 
various input devices such as a computer keyboard 106 
(connected by cable 108) and mouse or other control device 
110 (connected by cable 112 to the computer 102). 

A color video camera 114 is located approximately 48 
inches directly above the piano keyboard, and oriented to 
produce an unobstructed, upright image of the keyboard 
across the top of the frame, with the low notes on the left, 
A connecting cable 116 couples to the computer 102. The 
prototype is a cheap ($100) camera based on the NTSC 
standard and produces a satisfactory picture, although a 
suitably fast digital camera would give a superior image. 
The focal length of the camera lens is chosen to yield an 
image of the piano keyboard of the size described under 118. 
The camera support in the preferred embodiment is a 
tripod-like structure with one leg resting on each of the small 
platforms existing at the ends of most piano keyboards, and 
the third leg being weighted at the bottom and adjustable in 
length and angle, rests at a convenient position on top of the 
piano. This stmcture also supports the lamp, and optionally 
the computer monitor. 

A computer monitor 118 is provided for viewing by the 
user. Suitable specifications are an SVGA monitor at 1024x 
768 resolution, 16-bit color (although the application would 
also work with 8 -bit paEetized color,) and a 17-inch diagonal 
with 75 Hz refresh rate (CRT monitor) or 15-inch diagonal 
with 60 Hz refresh rate (flat panel monitor). Built-in speak- 
ers are a useful option. The monitor 118 is connected by 
cable 120 to the computer 102. 
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The video image of the piano keyboard appears at the Default settings for the chosen piece are loaded into the 
bottom of the monitor display, and spans the entire width of remaining Sel-Up-Screen parameters. The first of these 
the screen.At 1024x768 resolution it is necessary to truncate (block 146) is a yes/no pair of check boxes indicating 
the keyboard in the video image to make the remaining keys whether the audio feature is enabled. The default is "No", 
appear sufSciently large. The top 12 and the bottom 3 keys s The user next moves on to select a Start Point (block 14«), 
ate omitted. On the rare occasions when these extreme keys where the default is Measure 1, and an End Point, where the 
are played, their use is indicated by a special coding in the default is the last measure of the selected piece. The next 
animation. Although this arrangement is satisfactory, a option U a check box where one can choose to repeat (btodc 
larger monitor with a horizontal resolution greater than 1234 150) the selected measures. n,e default is unchecked, 
would accommodate the full keyboard image. A digital lO Checking the box causes the anmiaUon to cycle mdeflnilely 
monitor is also an option. TTie prototype monitor is a flat '^e selected measures, with a bnef pause before 
panel display resting on top of a closed grand piano. Amore ^^'^^ V ! fu' ^'''°^r', 
universally adaptable arrangement would replace the manu- option (block 152), which defaults to the number of beats 
facturer's stand with one designed to rest on the piano's P« ^^^^ mdicated on the score. Another choice "J the 
existing music stand. Alternatively, the monitor could be is Set-Up S^een ,s the degree of tempo variation (block 154)^ 
mountSl on the frame that supports the video camera and its P^^^V- ^ ha* ^'^ oppottumty to select advanced 
. options. The above features of the Set-Up screen are dis- 
, . . , J . . cussed in greater detail under FIG. 3. When finished, the user 

Other types of suitable display devices include a touch ^^^^ .j^^^^,, ^^^^j^ ^^^^ ^-^^ S^^.^p screen and 

screen monitor, and a virtual reahty device. Another option ^^^.^^ ^^^^^^ ^^^^^ 

is a specially designed fiat panel, or ti ing of flat panel ^^^.^^ ^^^^^ ^ ^ fmi-screen display where 

monitors placed m contact with an actual keyboard. ^^.^^^.^^ ^^.^^^ ^^^^^ ^^^^ p^^^ 

A musical keyboard 122 is shown in the drawmg of FIG. ^ -^j^^ -^^g^ t^e keyboard appears. Across the top is a 

1. The keyboard is preferably a relatively stationary monitoring and control bar containing various buttons and 

keyboard, such as a piano or organ. However, the invention ^^^^^ jj^^^ opening the Action Screen, if (block 158) 

may be applied to other instruments where the physical keyboard video has not been activated, the user is 

positioning of the hands relative to keys, valves or other prompted (block 160), to turn on the camera and lamp, and 

actuatable element may be imaged. ^^^^^ frame-grabber software. He then sizes and posi- 

A lamp 124 of some sort is often necessary to eliminate tjong video image (block 162) to bring it into registration 

shadow between tbe black keys when room lighting is vvith a dummy image of the keyboard across the bottom of 

predominantly from one side of the keyboard. A lamp can ^^i^ screen. (In future appHcations, some or all of these video 

also eliminate the shadow cast on the keys by the user when initialization tasks may be performed automatically by 

the room lighting is predominantly behind his back. A software.) 

forty-watt fluorescent lamp mounted about five inches above q^^^^ y^^^^ keyboard is displayed in the 

the back of the keyboard provides uniform illumination of proper position, the user notes whether the animation is 

the keys for the video camera. A standard 40-watt fluores- running. If (block 164) animation is running, the program 

cent tube is the same length as a standard piano keyboard. automatically clears any leftover activity (block 200), 

The lamp is customized to minimize its bulk and to direct its pauses about one second and resumes animation at the 

light downward. currently enabled start point (block 202). If at block 164 the 

A more practical alternative to the fluorescent lamp, but animation is not running, the user clicks the "Start" button 

one providing somewhat less-uniform illumination, is an (block 166), which launches the animation or, if the anima- 

appropriately coUimated spotlight 126 mounted above and a lion had been paused, clears the leftover sprites and after a 

little forward of the camera. Other lighting arrangements arc one-second delay, resumes the animation at the current start 

possible as well, as long as, day or night, they provide the point. Whfle the animation is running, the "Now playing 

dominant source of light on the keyboard and a satisfactory 45 Measure X, Beat Y" edit boxes on the toolbar are automati- 

video image. cally updated with current values for X and Y. Gukled by the 

FIG. 2 is a user flow chart illustrating the steps of human animation and the adjoining video image of his hands upon 

interaction with the graphic animation and live video image. the keys, the user plays the instrument, (block 168). If (block 

Dashed lines, other than those outlining the three screens, 170) at any time he chooses to pause the animation, he clicks 

indicate decisions or functions that are performed automati- 50 (block 204) the "Pause" button, and all moving sprites freeze 

cally by software, and do not require any response from the in place. If (block 206) he wishes to resume, he clicks (block 

user When starting the system (block 140), first to appear on 208) the "Start" button, the sprites unfreeze, and motion 

the monitor is the Opening Screen (block 141). This is a resumes where it left off. If he chooses not to resume, he 

full-screen display with some artwork and a welcoming decides (block 180) whether to quit the program, or to 

message accompanied by an audio clip of piano music. 55 activate a return (block 186) to the Set-Up Screen (block 

The user clicks "Start" (block 142) to hide the Opening 143). 

Screen and advance to the Sel-Up Screen (block 143), which Should the user become dissatisfied with the Set-Up 

is shown in detail in FIG. 3. This full-screen display pro- parameters (block 172), he can click the "Change settings" 

vides access to a library of piano pieces, and contains button on the toolbar, which hides the Action Screen and 

various track bars and edit boxes where one chooses the 60 returns the Set-Up Screen where the initial selections are 

set-up parameters governing the animated presentation of still displayed, awaiting any changes. Should the user oth- 

the selected piece. After a piece is chosen (block 144), a erwise become dissatisfied with his own performance (block 

coded version of the score, caUed the Piece File, is loaded 174), he can choose to jump back to the beginning of the 

into memory from the hard drive. The Piece File for a typical current measure (block 188), or jump to some other measure 

classical composition is a few tens of kilobytes, so the entire 65 and/or beat (block 190). In the "Jump to Measure X, Beat Y" 

file fits easily into RAM. Disk reads and the associated edit boxes, the Y defauU remains at zero. If a number for Y 

delays, therefore, are unnecessary during performance. is entered manually it persists until removed manually. The 
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X default is automatically updated to the measure in play, to that for the Start Point choice, with a similar trap for 

unless a number for X has been entered manually (block cut-of-range choices. If the same number is entered in both 

192), in which case the manual entry locks in (block 194) the Start Point and End Point boxes, the animation will start 

until the lock is released (block 198) by clicking the "Jump at the beginning of the selected measure, and stop at the end 
to Measure X, Beat Y" button (block 196). This button 5 of that measure, 

returns control to block 200 where old sprites are cleared. The next option is a check box 336 where one can choose 

and animation resumes (block 202) at the X, Y Start Point to repeat the selected measures. The default is unchecked, 

indicated in the button's edit boxes. Arrival at block 200 by Checking the box causes the animation to cycle indefinitely 

any other path causes the Start Point to revert to the through the selected measures, with a brief pause before 
beginning of the measure selected on the Set-Up Screen. lO ^^^^ restart. Next encountered is the "Set the Metronome^ 

Tf/ui 1 I'^icx^u c jn • ♦ 1 * J *k c * TT c option, a trackbar 338 that defaults to the number of beats 

If (block 176) the End Pomt selected on the Set-Up Screen • . ■ * ^ .u • -m. 

. ^ , , 1. J • J , *• /ui 1 per minute indicated on the piano score. The user can 

has not been reached, animation and play continue (block ^ ^ j r i. l • *u i*j u i j j- -*u * 

1 /:o\ Tf «i. J u u u A 4U ' *' 1 override the default by moving the slider back and forth to 

168). If the end has been reached, the animation area clears, ^^u* •« -i i-^erry 

j 1 • u 1 • *u un *» u selcct any integer between 10 beats per minutc and 125% of 

and if (block 178) there is a check in the "Repeat box on «u j r u -mf * a iu 
*u o * IT c * 1 * » ui 1 A i<; the default. The metronome setting IS displayed 342 to the 

the Set-Up Screen, control returns to block 202, and after a • . , ^ * i u a u • 1! -i- %u «u 

. . ^ ' , . Tf • u 1 ■ *u right of the trackbar. A user who is unfamihar with the 

1 -second pause, the cycle repeats. If there is no check in the l * j • n n i ♦ * k * iU «f 

"Repeat" Lx, .he use? decidL (block 180) to click either the r"beCrSu ^^^^^^^ 

"Change settings" button (block 186) and return to the ^ • • o it c • .u j r 

Set-Up Screen, or the "Exit" button (block 182) and end the ^h^ Set-^p Screen is the degree of 
• /ui 1 io>i\ 9n tempo vanation. Many musical scores include notation tor 

session (block 184). . , • j * r ■ i 

^ . ^ ^ , ^ rr ^ r « . Speeding Up aud slowing dowH the tempo from thc nommal 

Refernng to FIG. 3, the Set-Up Screen 143 follows the metronome value. The magnitude of these variations is 

Opening Screen. Here the user chooses a piece to play and ^^^-^ ^^istic interpretation. The preferred embodiment 

sets the parameters of the animated display to suit his taste invention errs on the high side for the magnitude of 

and level of advancement. He can choose to work on a variations programmed into the Piece File. Tempo 

subsection of the piece, set various audio opUons, etc. variations, when large, can present difficulty to a user just 

The piece selection tools 308 allow the user to browse the beginning to work with a new piece. The trackbar 340 lets 

library of choices stored in RAM and/or other appropriate the user select the magnitude of the tempo variations 

medium such as CD ROM. Clicking the "by title"' button between 0 and 100% of the values programmed in the Piece 

310 opens a window with library selections listed alphabeti- File. The selection is displayed 344 to the right of the 

cally by name of piece. The letters A, B, C . . . across the top trackbar. A setting of 0% produces a steady metronome 

of the window can be clicked to jump to pieces beginning tempo with no variation at all. The default is 30 percent, 

with that letter, or the list can be scrolled. The "by com- Whatever the trackbar setting, the tempo variation manifest- 

poser*' button 312 works similarly. Box 313 is available for ing in the animation also scales with the metronome setting, 
typing in key words in combination with Boolean operators. 3^ j^^^ "Advanced" button 346 brings up a window with 

Clicking the "Search" button 314, opens a window with the additional options. These can include recording and playing 

search results. Whether chosen by browsing or searching, ^ack the performed measures, optionaUy interleaved with a 

the name of the selected piece 316 appears on the Set-Up recording of those same measures as played by a profes- 

screeo after the choice is clicked and the browse or search gjonal artist. The playback can be heard through speakers, or 

window is closed. player-piano actuation of the keys themselves. Further pos- 

The audio options 318 can be ignored by the user. The sib ili ties are options for overriding Piece-File programming 

default is no audio, and a check appears in the "No" box 322. for fingering, tempo variations, and loudness, and for saving 

The remaining choices in 318 are grayed out. Or the user can the resulting customized version of the Piece File with a 

choose to hear the piece played by a professional musician "Save As" command. Other embodiments might also 
by clicking the "Yes" check box 320, and the subsequent 45 include various prior-art instructional enhancements to the 

choices in 318 become available. The first of these is animation display such as simultaneous viewing of the 

whether to listen to the entire piece at full tempo, check box musical score, with a cursor at the point of play. Also 

324, or, check box 326, to hear only the measures selected available might be alternative sprite forms with appeal to 

below in boxes 332 and 334 at the tempo resulting from the youngsters of various ages. For example, for very young 
choices in boxes 338 and 340. In the case of a check in 326, 50 children the sprites could resemble cartoon characters. Boys 

the pitch is maintained independent of the tempo by means age 7 to 14 might prefer rockets or bombs in a video-game- 

of prior art (U.S. Pat. No. 5,583,308). The user then chaoses like formal where points are awarded for correctly played 

whether to hear the professional version before he plays it, notes, and "catastrophic** explosions accompany mistakes, 

check box 328, and/or while he plays it, check box 330. "3D" enhancements to the animation may be incorporated 
Boxes 324, and 326 are mutually exclusive. Boxes 328 and 55 for added excitement for this group of users. Optional 

330 are not. sensors in the keys supply the required feedback for moni- 

The user next moves on to select a Start Point in the toring performance and correction of mistakes. Further 

"Begin at Measure" edit box, 332. The default is Measure 1, options include accompaniment for the user, such as the part 

He can type in a different measure number, or use the of the left hand alone, the primo or secondo of a duet, or the 
up-down arrows associated with the edit box to scroll to a 60 orchestral part of a concerto. 

number of his choice. Should he select a measure outside the If a piece has been selected, the "Next" button 348 hides 

range of the piece, the program displays an error message to the Set-Up Screen and brings up the Action Screen 157. If 

that effect, and pauses while he corrects the mistake. If the he has not chosen a piece, the user is prompted to do so. The 

new choice is within range, the error message will disappear Set-Up screen persists until a choice of piece is made, 
when he exits the edit box. Next he selects an End Point in 65 Additionally, if upon clicking the Next button, the selected 

the "End at Measure" edit box, 334 where the default is the ending measure number is not equal to cr greater than the 

last measure of the selected piece. The procedure is similar selected starting measure number, an error message requests 
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the conrectioD, and the program pauses while the change is refereoce to the identification numbers on the drawing by an 

made. The message disappears upon exit of the correspond- indication of "r" for red, *'b" for blue, and "g** for green, 

ing edit box, and the Next button can be clicked again. Having changed color to red a sprite continues its down- 

FIG. 4 shows a display in use of the device. The Action ward motion, sliding steadily behind the key image until the 
Screen 157 is where the animation takes place, guiding the 5 sprite has disappeared. Since the vertical length of the sprite 

user's hands to the proper keys at the proper time. Across the is created equivalent to the duration of its corresponding 

top of the screen is a monitoring and control bar 414 that note, the moment of its disappearance is when the key 

contains various edit boxes, buttons, and labels for display- should be released. (If the damper pedal is depressed the key 

ing and directing the progress of the performance. Control is can, of course, be released sooner.) More explicitly, the 

exercised by placement of the cursor over the appropriate vertical length of a sprite in pixels is the product of the sprite 

button, followed by a left click of the mouse. velocity in pixels per second and the duration in seconds of 

The name 416 of the selected piece is displayed above the the note it represents, as determined by the note type (quarter 

buttons for starting 418 and pausing 420 the animation. The note, eighth note, etc.,) the time signature of the piece, the 

most recenUy played beat number is displayed in box 424 metronome setting, and any departure from metronome 

beside its conesponding measure number in box 422. tempo. In the absence of tempo variations, a quarter-note 

Directly beneath these boxes are user writable edit boxes sprite, for example, is twice as long as an eighth-note spnte. 

428 and 432 that allow the user to enter measure and beat The downward velocity of all sprites is equal and 

numbers different from the current values, should he wish to constant, independent of metronome setting 146 or tempo 

click the "Jump to Measure" button 426 and restart play variations 154. At 1024x768 resolution, a 6 pixel jump every 
from some other position in the piece. The Measure edit box '^^ 56 milliseconds results in a suitably viewable sprite velocity 

428 defaults to the current number in the box 422 above. The of 107 pixels per second. (In the real world, the 

Beat edit box 432 defaults to zero, so clicking button 426 56-millisecond timer interval fluctuates with varying load on 

normally resumes play at the beginning of the current the CPU. To maintain a constant average sprite speed, the 

measure. Overrides for the measure and beat defaults can be size of each jump is adjusted to compensate for the concur- 

entered manually in the edit boxes by clicking the corre- rent deviation from the nominal 56-mUlisecond interval.) To 

sponding up-down arrows 430 and 434, or by typing in keep the display uncluttered, new sprites first appear when 

numbers with the computer keyboard. Once an override is Ihey are within just one beat of being played. Tempo 

entered in box 428 it remains fixed until the button 426 is variations, however, cause the time interval for a beat to 

clicked, after which default updating resumes. An entry in fluctuate. Consequently the distance from the keyboard 
box 432 remains unchanged until altered by the user. The ^° image to the initial sprite positions will vary with tempo. 

"Change Settings" button 436 hides the Action Screen and When tempo is changing from beat to beat, it is the interval 

displays the Sct-Up screen 143. The "Exit" button 438 of the previous beat that sets the distance of ncwly- 

closes the program. appearing sprites above the keyboard image. Of course, the 

Across the bottom of the screen is a dummy image spntes^ vertical lengths also fluctuate with tempo, so a 

(bitmap) of the piano keyboard. The dummy image serves as speed-up of tempo, for example, results m a vertical com- 

a template for sizing and aligning the live video image 412 prcssion of the visible animation. 

of the real piano keyboard. The live image is dragged over In the illustrated embodiment, blue sprites indicate notes 

the bitmap to hide all but a narrow (-3 mm) strip at the top to be played with the left hand, and green sprites indicate 

of the bitmap. The live image then is sized and dragged left notes to be played with the right hand. Color saturation of 

or right until the two images are in registration. The live the sprites indicates the loudness of the note to be played, 

image then can be dragged up the last few millimeters to with a gradation from light pastels for pianissimo, to bright, 

completely hide the bitmap image. Optionally, the alignment richly saturated colors for fortissimo. A black bottom edge 

procedure may be automated. on a sprite indicates that the damper pedal should be 
The animation activity takes place in the area between the 45 depressed when the sprite touches the key image. A black 

keyboard image 412 and the monitoring and control bar 414. lop edge on a sprite indicates that the damper pedal should 

Time is the vertical co-ordinate in this space. The horizontal be released when the last of the spnte disappears behmd the 

co-ordinate is distance along the keyboard, in one-to-one *^^y image. 

correspondence with the bordering keyboard image. The The design of the sprites can convey other information 
term 'horizontal' means generally parallel to the keyboard. 50 carried in musical notation as well. The rounded edges and 

Each key to be struck within one beat of time is designated full length of the sprites illustrated here represent legato, or 

by a colored "sprite", for example 446, directly above, and slurred notes played smoothly and evenly. Sprites having 

moving steadily downward toward the image of that key. slightly clipped ends represent portamento, or more delib- 

The distance in pixels from the top of the keyboard image to erate execution. Staccato is shown by more severe clipping, 
the leading edge of a sprite at any instant is the product of 55 significantly shortening and squaring the sprites to indicate 

the sprite velocity in pixels per second, and the time in notes that are abruptly disconnected. A martellato is repre- 

seconds remaining until the key is to be struck. A targeted sentcd by a sprite with a completely flat leading edge to 

key is played by the musician when the corresponding sprite signify a note to be "hammered". 

first touches the key's image on the screen. This point in Xo assist the user in identifying the key targeted by an 
time is enunciated by an abrupt change in color of the sprite, go approaching but still distant sprite, for example the b-flat 

for example, to red. Thus, like a skeet shooter taking a bead targeted by sprite 446fc, a small circular field 4486 (call it a 

on his target, the musician views each approaching note in "keylight") of the same color as the sprite, but fixed in 

advance of playing it. In the live image of his hands passing position immediately above the image of the key, appears 

over the keys, he sees his fingers converge with the sprites simultaneously v/ith the sprite. Within the key light is a bold 
for the strike. 65 Arabic numeral to indicate the finger to be placed on the key, 

FIGS. 4 through 6 include sprites of various color or 1 for thumb, 2 for index finger, etc. The keylight persists 

hatching. The designation of colors will be depicted with until contacted by the sprite, at which time the keylight 
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vanishes. Should another sprite target the same key before 
the previous sprite has cleared, the most recent keylight is 
superimposed on whatever sprite or earlier keylight occupies 
the keylight position. 

The illustration finds the animation just beyond the play- s 
ing of measure 5, beat 5 of the Chopin Nocturne in E-Flal. 
The measure and beat numbers for the most-rccently played 
notes are displayed in boxes 422 and 424. Sprites for notes 
already played 440/-, 442r, and 444r, have turned red and are 
sliding behind the keyboard image. The user's fingers are ^0 
still on the respective keys. Four sprites for about-to-be- 
played notes 446b, 450b, 454g, and 458g, are approaching 
the keyboard image from above. Their respective "key- 
lights" 44Sb, 4S2b, 45 6g, and 460^, are in place. 

Tempo variations have been set to zero and the metro- 
nome has been set at 60 beats per minute for this illustration, 
so the beat interval in this case is exactly one second. The 
time signature for this piece is 12/8, meaning that there are 
12 beats per measure, and an note gets one beat. Each of 
sprites 446b and 4505 represents an 8*'* note, and therefore ^ 
each has a vertical length equivalent to one second of time. 
These two sprites are a little less than one second from 
contact with the keyboard image. Their blue color signals 
that they are to be played with the left hand, and their 
keylight numbers designate the thumb for 450b and the little 
finger for 446b. 

Sprites 454^ and 458^ represent 16*'' notes. They are 
one-half second long, and their keylights and color indicate 
that they are to be played with the thumb and third finger of 
the right hand. Sprite 45Hg being closer to the keyboard 
image will be played first. It is scheduled for play on beat 
5Vi, about a quarter-second hence. Sprites 4466, 450fc, and 
4S4g are scheduled for play on beat 6, about three quarters 
of a second hence. 

FIG. 5 is a snapshot of the Action Screen about one- 
quarter second later than illustrated in FIG. 4. Here the 
action is just at the start of beat 5.5. All of the sprites of FIG, 
4 have moved downward, with sprite 444 having slid 
completely from view. Sprite 458r has jnst touched the 
keyboard image and changed color to red, indicating that 
now is the time to play the targeted a-fiat. The corresponding 
keylight 460 has vanished. A new sprite 464^ has appeared, 
its corresponding note now being within one beat of the 
keyboard image. Color-coding and the keylight for the new 
sprite indicate that it is to be played with the index finger of 
the right hand. 

FIG. 6 is a snapshot of the Action Screen one-half second 
later than illustrated in FIG. 5. Here the action is just at the 
start of beat 6. All of the sprites of FIG. 5 have moved 50 
downward, with sprites 440, 442, and 458 having slid 
completely from view. Sprites 446r, 450r, and 454r have 
just touched the keyboard image and changed color to red, 
indicating that now is the moment to play the respective 
b-flat, a -flat, and c keys. The corresponding keylights 448, 55 
452, and 456 have vanished. The black tops on sprites 446, 
450, and 464 signal the release of the damper pedal at the 
moment these sprites slip from view. 

TWo new sprites have appeared, since their corresponding 
notes are now within one beat of play. Sprite 4706 represents 60 
an S''* note, and its color and keylight 4726 indicate that it 
is to be played with the httle finger of the left hand. Sprite 
466g represents a dotted quarter note (three seconds in 
duration). Its color and keylight 468g indicate that it is to be 
played with the third finger of the right hand. The black 65 
bottoms on sprites 470 and 466 indicate that the damper 
pedal should be pressed back down when these sprites 



contact the keyboard image and turn red. (Technically, since 
the contact of the black-bottomed sprites with the keyboard 
image is simultaneous with the disappearance of the black- 
topped sprites, the damper pedal also has to be released and 
depressed simultaneously. This is not possible, but it is 
understood in this case that the pedal should be released, 
then immediately pressed back down.) 

The top of the long sprite 466 is initially only a short 
distance from the monitoring and control bar at the top of the 
screen. It would take only a small decrease in tempo to 
stretch the sprite beyond the lower edge of the bar. When this 
situation occurs, only the portion of the sprite between the 
bar and the keyboard image is visible. As the sprite scrolls 
downward, the top eventually emerges from behind the bar, 
and continues on to disappear behind the keyboard image at 
the proper time. 

FIGS. 4-6 represent less than one second of elapsed time 
out of about six minutes for the entire 35 measures of the 
nocturne at this reduced tempo. At normal tempo (130 beats 
per minute) the piece takes about three minutes. 

The foregoing cited references, patents and publications 
are hereby incorporated herein by reference, as if fully set 
forth herein. Although the foregoing invention has been 
described in some detail by way of illustration and example 
for purposes of clarity and understanding, it may be readily 
apparent to those of ordinary skill in the art in Hght of the 
teachings of this invention that certain changes and modi- 
fications may be made thereto without departing from the 
spirit or scope of the appended claims. 

What is claimed is: 

1. A method for prompting play of a musical score by a 
user of a musical instrument having a keyboard actuated by 
the user's hands, comprising the steps of: 

displaying a graphical representation of at least a portion 

of the musical score for viewing by the user, 
imaging in real time at least a portion of the keyboard and 

the said iiser's hands, 
displaying to the user at least a portion of the image of the 

keyboard and user's hands in correlation with the 

display of the graphical representation of the musical 

score, 

2- The method for prompting play of a musical score by 
a user of a musical instrument of claim 1 wherein the 
displaying is provided on a monitor. 

3. The method for prompting play of a musical score by 
a user of a musical instrument of claim 1 wherein the 
musical instrument is a keyboard instmment. 

4. The method for prompting play of a musical score by 
a user of a musical instrument of claim 3 wherein the 
keyboard instrument is a piano. 

5. The method for prompting play of a musical score by 
a user of a musical instrument of claim 3 wherein the 
keyboard instrument is an organ. 

6. The method for prompting play of a musical score by 
a user of a musical instrument of claim 1 wherein the notes 
are depicted by graphical animation which is registered in a 
horizontal manner to the key to be actuated. 

7. The method for prompting play of a musical score by 
a user of a musical instrument of claim 1 wherein the notes 
arc depicted by graphical animation which is registered in a 
horizontal manner to the key to be actuated and provided to 
the xiser prior to the time of actuation of the key as required 
by the mtisical score and progressing over time, culminating 
in an indication to the user that actuation of the key is then 
required by the musical score. 

8. The method of claim 7 wherein the graphical animation 
of the note is moved in a vertical direction. 
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9. The method of claim 8 wherein the motion in the 
vertical direction is linear in time, 

10. A method for prompting playing a musical score by 
the user of a musical instrument having a keyboard actuated 
by the user's hands, comprising the steps of: 5 

displaying the musical notes of the score to the user, 
wherein the notes are depicted by graphical animation, 
the depiction of the note being: 
registered in a horizontal manner to the key to be 
actuated, and 

provided to the user prior to the time of actuation of the 
key as required by the musical score and progressing 
over time, culminating in an indication to the user 
that actuation of the key is then required by the 
musical score, and 15 
displaying a real time image of the user's hands adjacent 
the keyboard. 

11. The method of claim 10 wherein the graphical ani- 
mation of the note is moved in a vertical direction. 

12. The method of claim 11 wherein the motion in the 
vertical direction is linear in time. 

13. The method of claim 10 wherein the culmination of 
the graphical display for a note consists of a juxtaposition of 
the image with the key to be played. 

14. The method of claim 10 wherein the graphical ani- 
mation of the note changes in size as a function of time. 

15. The method of claim 10 wherein the graphical ani- 
mation of the note changes in intensity as a function of time. 

16. The method of claim 10 wherein the graphical ani- 
mation of the note changes in orientation as a function of 
time. 

17. The method of claim 10 wherein the graphical ani- 
mation of the note changes in appearance as a function of 
time. 

18. The method of claim 10 wherein the graphical ani- 
mation of the note changes in color as a function of time. 

19. The method of claim 10 wherein the graphical image 
of the note is present to the user sufficiently prior to the time 
of actuation to permit the user to place their hands for timely 
actuation of the key. 
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20. The method of claim 10 further including an addi- 
tional graphical display adjacent to the key to be actuated. 

21. The method of claim 20 wherein the additional 
graphical display indicates the desired finger number to 
actuate the key. 

22. The method of claim 21 wherein the finger number is 
a numeric indication. 

23. The method of claim 20 wherein the additional 
graphical display is presented at the same time as the 
depiction of the note. 

24. The method of claim 20 wherein the additional 
graphical display is fixed in position with its lowest point in 
contact with the targeted key. 

25. The method of claim 10 further including the step of 
imaging a musical score and generating the graphical ani- 
mation therefrom. 

26. A method for displaying a musical score to a user of 
a musical instrument having a plurality of keys comprising 
the steps of: 

graphically depicting an object corresponding to a note to 
be played, the graphical position including a position 
relative to the key to be actuated, the graphical depic- 
tion of the note including a depiction of the method of 
attack of the play of the note, and 

a temporal dimension indicating both the time for actua- 
tion of the note and the duration of the activation of the 
note. 

27. The method of claim 26 wherein the graphical depic- 
tion further includes the loudness of play of the note. 

28. The method of claim 26 wherein the method of attack 
is selected from the group consisting of: staccato and slur. 

29. The method of claim 26 wherein the graphical depic- 
tion further includes an indication of an addition action to be 
performed by the user. 

30. The method of claim 29 wherein the additional action 
includes pedal actuation. 

31. A computer readable code for use in a system for 
aiding in play of a musical instrument for implementing the 
method of claims 1, 10 or 26. 
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